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and  the  Smallest  Perturbation  of  a  Sobmatrix  which  Lowers  the  Rank 
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Abstract 


Letr= 


A   B 
C  D 


be  a  partitioned  rectangiilar  matrix.  We  exhibit  perturbations  of  D  of  smallest 


norm  which  lower  the  rank  of  T.  We  apply  this  result  to  generalize  the  total  least  squares 
problem  Sx  =  b  when  only  a  rectangular  subset  of  the  data  [S,b]  is  considered  uncertain. 


1.  Introdactlon 


Let  T= 


A   B 
C  D 


be  a  partitioned  rectangular  matrix.  We  exhibit  perturbations  of  D  of 


smallest  Frobenius  or  two  norm  which  cause  T  to  h^ve  a  lower  rank.  This  generalizes  a 
recent  result  of  Stewart  [2]  who  found  the  smallest  rank  lowering  perturbation  of  D  when 
T=[ClD].  It  may  be,  of  course,  that  the  rank  of  T  is  independent  of  D,  as  in  the  2  by  2 
example 


T  = 


0  1 

1  D 


which  is  of  full  rank  for  all  scalars  D.  In  such  cases  our  method  will  report  that  no 
perturbation  of  D  exists.  In  such  cases  a  small  change  in  the  matrix  can  cause  a  large  change 
in  the  answer.  If 


n€)  = 


-€      1 

1     D 


then  D  must  equal  t~Mo  lower  the  rank.  As  e  decreases  to  0,  t"^  grows  to  infinity,  until  at 
€=0  no  rank  lowering  perturbation  of  D  exists.  Thus  the  problem  of  determining  the  smallest 
rank  lowering  perturbation  of  D  can  be  quite  ill-conditioned.  This  is  in  marked  contrast  to 


the  classical  problem  of  finding  a  smallest  rank  lowering  perturbation  to  the  matrix  as  a 
whole.  In  this  case  the  answer  is  the  smallest  singular  value  of  the  matrix,  and  it  can  change 
no  more  than  the  matrix  itself  changes  (measured  in  the  2-norm). 

We  apply  this  result  to  generalize  the  procedure  of  total  least  squares  (TLS)  as 
described  in  [1].  TLS  arises  when  trying  to  solve  a  least  squares  problem  Sx=b  where  5  has 
more  rows  than  colimins.  In  traditional  least  squares,  one  considers  the  entries  of  S  to  be 
exact  and  the  entries  of  b  to  be  contaminated  with  noise,  and  one  interprets  the  residual  norm 

min  ||5x-*||  =     min     ||r||   , 
(where  ^(5)  is  the  span  of  the  columns  of  5  and  ||-||  is  the  Euclidean  vector  norm)  as  a 
measure  of  this  contamination.  In  TLS,  one  considers  the  entries  of  both  5  and  b  to  be 
contaminated  with  noise,  and  interprets  the  value  of 

min       |I[£,r]||^ 

b+riR{S+S) 

(where  ||H;r  is  the  Frobenius  norm)  as  a  measure  of  that  contamination.  If  the  noise  in  some 
components  of  S  and  b  are  to  be  weighted  differently,  one  can  compute 

Tmn''\\WjE,r]W2\\r 
where  Wj  and  Wj  are  nonsingular  diagonal  weight  matrices.   Golub  and  Van  Loan  show  that 

the  value  of  this  minimum  is  given  by  (Jroini^ii^M^i)^  ^c  smallest  singular  value  of 

W^[S,b]W2- 

We  generalize  this  TLS  procedure  to  the  case  where  only  some  rows  and  columns  of  5 
and  b  are  contaminated.  If,  for  example,  only  the  last  m  rows  of  S  and  b  and  the  last  n 
columns  of  S  are  contaminated,  applying  our  result  to 


A   B 
C  D 


T  =  Wj[S,b]W2  = 

where  D  is  m  by  n  +  1  provides  a  measure  of  the  contamination  in  these  entries.  Another 
interpretation  of  this  result  is  as  a  solution  of  the  constrained  total  least  squares  problem, 
since  by  considering  A  and  B  to  be  exact,  they  provide  constraints  on  the  solution  vector. 


2.  Main  Result 

There  are  a  number  of  cases  in  our  result  depending  of  the  ranks  of  submatrices 
obtained  in  a  sequence  of  transformations  performed  on  T.  Therefore,  we  present  our  result 
as  an  algorithm  for  determining  the  smallest  rank  lowering  perturbation  of  D;  comments 
proving  the  correctness  will  be  inserted  in  the  algorithm. 

Our  algorithm  will  make  frequent  use  of  the  singular  value  decomposition  (SVD), 
which     we     will     write     as     follows:     A  =  i/2V^,     with     U     and     V     orthogonal     and 

2  =  diag(cri a^),  a{^  ■  ■  ■  aa^^O.  One  can  read  the  rank  of  A  directly  from  its  SVD:  it 

is  the  number  of  nonzero  a,.  In  terms  of  the  SVD,  it  is  will  known  that  a  smallest 
perturbation  of  A  which  lowers  its  rank  to  r  is  -t/diag(0.  .  .  .  ,0,(t,+i ^k)^^^  where  r 

k 

must  be  less  than  or  equal  to  ifc.  The  Frobenius  norm  of  this  perturbation  is  (  5)    o'?)^^  and 

/-r+l 

its  two  norm  is  cr^^^.  Our  algorithm  will  also  make  use  of  row  compressions  and  column 
compressions,  which  can  be  computed  in  several  ways.  Finding  a  orthogonal  matrix  U  which 
column  compresses  a  matrix  C  to  the  left  (resp.  to  the  right)  mejuis  finding  U  such  that  CU 
has  r£uik(C)  linejirly  independent  columns  as  its  leftmost  (resp.  rightmost)  columns,  and  the 
other  columns  zero.  It  can  be  computed  either  by  doing  a  QR  decomposition  of  C^  or  by  a 
singular  value  decomposition  of  C  [1].  Similarly,  row  compressing  C  upwards  (resp. 
downwards)  means  finding  a  orthogonal  U  such  that  UC  has  rank(C)  linearly  independent 
rows  at  the  top  (resp.  bottom)  and  the  other  rows  zero. 

Our  approach  depends  on  a  result  of  Stewart's  which  we  state  here: 

Algorithm  1:  [2]  Given  r=[Cl£)],  C  an  n  by  m^  matrix  and  D  an  n  by  m^  matrix,  compute  a 
smallest  perturbation  of  D  which  lowers  the  rank  of  T  to  r<n  (if  one  exists). 

1)  Find  U  to  row  compress  C  upwards.  Let  c  ■■  rank(C)  be  the  number  of  nonzero  rows  of 
UC.  Write 


Ci  D, 


0    Z), 


UT  = 
where  D^  is  c  by  m^  and  D2  is  n-c  by  m2. 

2)  If  c>r,  the  rank  of  T  is  greater  than  r  independent  of  D,  since  the  first  c  rows  of  f/C  are 
independent.  Terminate  the  algorithm. 

3)  (At  this  point  c^r.)  Qearly,  rank(r)  =  c  +  rank(D2).  so  the  problem  reduces  to  finding  a 
smallest  perturbation  of  D-^  which  lowers  its  rank  to  r—c.  This  problem  is  solved  by  the 
SVD.  Let  UqIdV'^)  be  the  SVD  of  Di,  and  let  d2  =  rank(£>2).  If  c  +  dj-^r,  the  rank  of  T  is 
already  less  than  or  equal  to  r.  Otherwise  the  smallest  perturbation  of  D  that  lowers  the  rank 


of  r  to  /■  has  Frobenius  norm  (    2      ^})^'^'  ^^  equals  U^ 

/-r-c+l 


0 

8D2 


,  where  bDj  is  the  smallest 

perturbation  of  Di  which  lowers  the  rank  of  D2  to  r—c.  Terminate  the  algorithm. 
Note  that  the  rank  of  T  must  lie  between  c  and  c  +  min(n-c,m2)  for  all  D. 
Now  consider  the  more  general  case  where 


T  = 


A   B 
C  D 


where  A  is  m^  by  n^  and  D  is  m2  by  n2.  We  assume  mi^^n^;  other  consider  T^.  To  illustrate 
how  we  might  reduce  this  general  case  to  the  case  covered  by  Algorithm  1,  let  us  consider  the 
special  case  where  A  is  of  full  rank  m^,  and  in  particular  let  us  assimie  that  the  first  m^  by  m^ 
submatrix  of  A  is  nonsingular.  Accordingly  partition 


T  = 


Ai  A2   B 
Ci  C2  D 
where  i4i  is  m^  by  m^  and  nonsingular.  Now  premultiply  T  by  the  nonsingular  matrix 


0 
-C,A7^   I 


I 


to  obtain  the  matrix  of  equal  rank 


Ai  A2  B 

0    Ci-CiAfUj  D-C^Al^B 

At  this  point,  since  A  ^  has  full  rank  m^,  it  is  dear  that 

rank(r)  =  m^  +  rank[C2- CjAf^Aj  \D-C/i'^^B]   , 
so  that  the  smallest  perturbation  of  D   that  lowers  the  rank  of  T  to  r  is  the  smallest 

perturbation  of  D  that  lowers  the  rank  of  [C2-C]AfU2  l^~CiAf^fl]  to  r-nti  (if  r^m.^; 

otherwise  no  such  perturbation  of  D  exists).  This  subproblem  is  solved  by  algorithm  1. 

Computing  C^y^B  can  be  quite  ill-conditioned  if  A  ^  is  nearly  singular;  as  illustrated  by  the 

example  in  the  introduction,  it  is  unavoidable. 

The  following  algorithm  generalizes  this  approach  to  the  case  where  A  is  not  of  full 
rank: 


Algorithm  2:  Given  T= 


A   B 
C  D 


where  A  is  m^  by  nj^  and  D  is  m2  by  nj,  compute  a  smallest 
perturbation  of  D  which  lowers  the  rank  of  T  to  r<min(mi+m2,«i+«2)  ('^  0°^  exists). 

1)  Find  an  orthogonal  Uy  which  row  compresses  A  upwards  and  an  orthogonal  Ui  which 
column  compresses  the  resulting  rows  lefUvard.  In  other  words  U-^Ui  has  a  nonsingular 
matrix  A^  in  its  first  a  «■  rank(A)  rows  and  columns,  and  the  other  rows  and  columns  are 
zero.  Then  transform  and  partition  T  as  follows: 

where  the  first  block  row  [AjO[flJ  and  first  block  column  [A[|0|C[]^  may  be  null  if^a  =  0, 
where  the  second  block  row  [0|0lfl2]  niay  be  null  if  a^m^,  and  where  the  second  block 
column  [0|0|C2l^  may  be  null  if  a  =  ni. 

2)  If  a  =  m^  so  that  the  second  block  row  is  null,  we  have  reduced  the  problem  to  the  special 
case  discussed  above.  Premultiply  Ty  by 


C/i    0 

U2    0 

Ai    0    fli 

0  i„ 

•  T  ■ 

0      ^n 

0     0    52 
Ci  C,    D 

-CM'  i„. 


to  get 


r,  = 


i4i    0 


fl, 


0    Cj  D-Cii4f^fli 


so  the  rank  of  T  is  rankCTj)  =  a  +  rank[C2|£)-Ci/4f  ^flj.  Let  <r  —  rank(C2).  Applying 
Theorem  1,  we  see  that  if  if  r<a  +  c,  no  pertxirbation  of  D  exists  which  lowers  the  rank  to  r. 
Otherwise,  apply  Theorem  1  to  compute  the  smallest  perturbation  of  D  that  makes 
rank[C2|D  — C^Af  ^5j=r— a.  Terminate  the  algorithm. 

3)  Now  a<m^  so  that  Bj  in  7^  is  nonnull.  Find  an  orthogonal  U^  which  row  compresses  ^2 
upwards  and  an  orthogonal  U^  which  column  compresses  the  resulting  rows  leftward.  In  other 
words  U-fi-^Ui,  has  a  nonsingular  matrix  ^21  'i*  *ts  first  b  ■  rank(fl2)  rows  and  columns,  and 
the  other  rows  and  columns  are  zero.  Then  transform  and  partition  T^  as  follows: 


7-3  = 


h    0     0 

^"^    ° 

0    {/j    0 

•7-1- 

1 

0    (/, 

" 

^1  ^  ^U    ^12 

0  0  ^21     0 

0  0  0      0 

Ci  C2  D,    D2 


Since  the  third  block  row  is  zero  it  does  not  contribute  to  the  rank  so  we  drop  it.  Also,  we 
swap  the  first  and  second  block  rows  and  move  the  third  block  column  to  the  first  column  to 
get  the  matrix  of  equal  rank 

7-4  = 

Let  c  -  rank(C2).  Note  that  in  T^  the  first  block  row  and  block  column  may  be  null  if  3  =  0, 
and  the  second  block  row  and  block  column  may  be  null  if  a-0,  and  the  last  block  column 
may  be  null  if  b  =  n2. 

4)  If  fe  =  n2,  so  that  the  last  block  column  of  T^  is  null,  then  rank(r)  is  independent  of  D  and 
we  are  done.   This  is  because  T^  has  rank 


Bn 

0 

0      0 

Bn 

^1 

0    B^ 

Di 

Ci 

C,   D, 

K 

0 

oV 

fill 

^1 

0 

0 

Ci 

^A 

h        0    0 
rank  0         I^    0     ■  T^    =  rank    B,,  A^    0       =  a  +  b  +  c 

which  is  independent  of  D.  Terminate  the  algorithm. 

5)  Now  b<n2  so  Di  is  nonnull.  If  a  +  &  =  0,  so  that  the  first  two  block  rows  and  block  columns 
of  T^  are  null,  we  have 

T,  =  [C2P2] 
which  is  the  case  covered  by  Theorem  1.  Then  rank(r)  is  at  least  c  =  rank(C2)  for  any  D,  and 

if  r^c  Theorem  1  provides  a  perturbation  SZ>2  of  D2  of  smallest  norm  which  reduces  the 

rank  of  T^  to  r.  A  smallest  pertiirbation  ci  D  is  in  turn  given  by  [0|5D2]^r-  Terminate  the 

algorithm. 

6)  Now  b<n2  and  a  +  b>0.  This  is  the  special  case  illustrated  earlier,  since  the  a  +  b  by  a  +  b 
matrix 

^21     0 

fill    '^1 

is  nonsingular.  Therefore  we  premultiply  T^  by 


^a  +  b 

0 

-[^ilcj 

fi21     0 
fill    '^l 

-1 

getting 


521    0     0 

0 

fill  ^^1    0 

fili 

0     0    C2  D2- 

-  c,a:'b^ 

T<  = 


so  that  the  rank  of  T  is 

rank(r)  =  rank(rj)  =  a  +  b  +  rank([C2|£>2  -  CiA^^B^])   . 
Applying  Theorem  1  we  see  that  rank(r)  a  a  +  Z>  +  c  for  any  D,  so  our  problem  has  a 

solution  only  ii  a  +  b  +  c^r.  Proceeding  as  in  Algorithm  1,  Let  U^  row  compress  C2  upwards 

and  partition 


Us[C2p2-CA:'B^]=     0   ^^^_j^^ 

where  C21  has  full  rank  c.  Let  8D22  be  a  smallest  perturbation  of  D22  that  lowers  the  rank  of 
D21—X2  to  r-a-b-c.  Then  a  smallest  norm  lowering  perturbation  of  D  is  given  by 


[0|^5^[8D22p'  • 
Terminate  the  algorithm. 

Note  that  we  have  also  shown  that 

rank(r)  a  a  +  ft  +  c  «■  rank(A)  +  rankCBj)  +  rankCCj) 
independent  of  D,  where  we  interpret  any  of  a,  ft  or  c  as  zero  if  the  corresponding  submatrix 

is  null.  In  addition,  a,  b  and  c  may  be  determined  by  only  using  unitary  transformations  on 

T,  whereas  determining  the  smallest  rank  lowering  perturbation  of  D  requires  nonunitary 

(and  possibly  quite  ill-conditioned)  transformations. 
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